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Abstract 

Acquisition  data  lay  the  foundational  role  for  decision-making,  management,  and  oversight  of 
the  weapon-systems  acquisition  portfolio  for  the  Department  of  Defense.  How  to  effectively 
and  efficiently  spend  these  dollars  has  been  a  top  priority  for  the  Better  Buying  Power 
initiatives  led  by  the  Office  of  the  Secretary  of  Defense  (OSD)  and  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics.  The  OSD  asked  RAND  to  help  identify 
how  available  data  can  help  assist  defense-acquisition  decision-making.  In  particular,  we 
documented  factual  information  on  21  information  systems  that  contain  acquisition  data.  This 
builds  on  our  earlier  work  (Riposo  et  al.,  2015,  Issues  With  Access  to  Acquisition  Data  and 
Information  in  the  Department  of  Defense:  Policy  and  Practice,  RAND  RR-880;  and 
McKernan  et  al.,  2016,  Issues  With  Access  to  Acquisition  Data  and  Information  in  the 
Department  of  Defense:  A  Closer  Look  at  the  Origins  and  Implementation  of  Controlled 
Unclassified  Information  Labels  and  Security  Policy,  RAND  RR-1476)  by  exploring  in  more 
detail  the  data  that  support  decision-making. 

Introduction 

Acquisition  data1  lay  the  foundation  for  decision-making,  management,  and  oversight 
by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 


1  Acquisition  data  are  vast  and  include  such  information  as  the  cost  of  weapon  systems  (both 
procurement  and  operations),  technical  performance,  contracts  and  contractor  performance,  and 
program  decision  memoranda.  These  data  are  critical  to  the  management  and  oversight  of  the  $1 .5 
trillion  portfolio  of  major  weapon  programs  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (OUSD[AT&L]). 
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of  the  weapon-system  acquisition  portfolio  for  the  Department  of  Defense  (DoD).  Acquisition 
data  help  to  inform,  monitor,  and  achieve  several  DoD  objectives,  including 

•  promoting  transparency  in  spending 

•  understanding  and  achieving  cost  control 

•  visualizing  the  distribution  of  defense  spending 

•  achieving  small-business  goals 

•  identifying  and  preventing  fraud,  waste,  and  abuse 

•  conducting  analyses  for  improved  decision-making 

•  compiling  and  tracking  items  in  various  processes 

•  archiving  decisions 

It  is  critical  for  personnel  managing  acquisition  execution  and  oversight  to  know  what 
data  resides  within  DoD  as  well  as  what  questions  can,  or  cannot,  be  answered  with  that 
data  (Table  1). 

Table  1.  Acquisition  Data  Can  Answer  Some  Defense  Questions,  and  Not  Others 


(RAND) 

Acquisition  Data  Can  Answer 

Acquisition  Data  Cannot  Answer 

How  much  do  we  spend  acquiring  weapons 
and  services  in  DoD? 

Where  can  we  access  certain  technologies? 

How  long  does  it  take  to  field  a  weapon 
system? 

What  key  suppliers  may  need  help  given 
spending  decreases? 

What  are  the  key  costs  of  what  we  buy  for 
DoD? 

How  well  does  spending  align  with 
requirements? 

Who  do  we  buy  from? 

What  value  is  gained  from  spending? 

Where  is  our  spending  geographically 
located? 

How  is  the  health  of  the  defense  industrial 
base? 

Why  are  costs  and  schedule  increasing? 

What  is  the  quantity  adjusted  cost  growth  for  a 
specific  acquisition  program? 

How  competitive  is  our  supply  base? 

How  do  the  acquisition  schedules  of  a  large 
number  of  acquisition  programs  compare  over 
time? 

How  well  does  DoD  meet  small  business 
goals? 

What  is  the  workload  of  various  parts  of  the 
acquisition  workforce  (e  g.,  contracting)? 

What  can  we  learn  from  past  acquisition 
failures/successes? 

Which  government  and  non-government 
personnel  are  working  specific  acquisition 
programs? 

How  to  effectively  and  efficiently  spend  taxpayer  dollars  allocated  to  the  Department 
of  Defense  has  been  a  top  priority  of  the  Better  Buying  Power  (BBP)  initiatives  led  the  Office 
of  the  Secretary  of  Defense  (OSD)  and  the  USD(AT&L).  In  BBP  2.0,  the  USD(AT&L) 
specifically  acknowledged  the  need  to  streamline  decision-making  by  “promptly  acquiring 
relevant  data  and  directing  differences  of  opinion  to  appropriate  decision-makers.  Our 
managers  cannot  be  effective  if  process  consumes  all  of  their  most  precious  resource — 
time”  (Kendall,  2013,  p.  2). 
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Currently,  much  weapon-system  acquisition  data  is  collected  based  on  policy 
directive,  congressional  reporting,  and  the  need  to  meet  USD(AT&L)’s  statutory  authorities. 
These  information  requirements  largely  reside  in  the  Department  of  Defense  Instruction 
(DoDI)  5000.02  (2015).  This  data-management  strategy  fails  to  address  the  complete 
managerial  prerogatives  of  the  USD(AT&L)  and  the  Better  Buying  Power  initiatives. 
Additionally,  siloed  reporting  of  acquisition  data  may  not  fully  support  the  USD(AT&L) 
decision-making  processes.  Data  requirements  have  generally  been  developed  from  a 
particular  functional  perspective  resulting  in  a  data  “ecosystem”  characterized  by  individual 
collections  of  data  that  are  functionally  stovepiped  and  disjointed,  each  with  different  rules 
for  collection,  retention,  and  access. 

Approach 

In  earlier  work  (Riposo  et  al.,  2015;  McKernan  et  al.,  2016),  we  identified  the  issues 
associated  with  managing  and  sharing  Controlled  Unclassified  Information  (CUI)  within  the 
DoD.  In  this  analysis,  we  examine  issues  with  managing  and  accessing  the  sources  of  that 
data.  Specifically,  the  OSD  asked  us  to  consider 

•  What  data  are  available  to  help  assist  in  defense  acquisition  decision¬ 
making? 

•  Where  do  acquisition  data  reside? 

•  Who  can  access  the  information? 

•  Can  we  get  access  to  these  data  for  acquisition-related  purposes? 

To  answer  these  questions,  we  held  targeted  discussions  with  acquisition  information 
system  managers,  supplemented  these  discussions  with  reviews  of  official  policy 
documentation  and  other  open  sources  on  the  information  systems  and  their  contents, 
reviewed  literature  on  master  data  management  to  understand  practices  in  commercial  data 
management,  and  augmented  our  findings  with  RAND  knowledge  of  using  these  data 
systems.  Through  these  methods,  we  accomplished  four  tasks. 

What  are  the  major  weapon  system  acquisition  data  domains?  We 

accomplished  this  task  by  reviewing  various  federal-wide,  OSD-wide,  and  Service-level 
information  systems  and  their  data  elements  in  order  to  identify  where  the  data  that  supports 
current  information  requirements  in  DoDI  5000.02  reside.  We  focused  first  on  a  broad  look 
at  the  enterprise  acquisition  landscape  as  a  whole,  then  particularly  on  sources  of 
acquisition  information  that  support  the  USD(AT&L)  through  the  Defense  Acquisition 
Executive  Summary  (DAES)  and  Defense  Acquisition  Board  (DAB)  secretariat,  Director, 
Acquisition  Resources  and  Analysis  (D,  ARA).  Our  sponsor,  deputy  director  of  Enterprise 
Information,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics,  Acquisition  Resources  and  Analysis  Directorate,  provided  the  list  of  information 
systems  to  examine  for  this  analysis. 

What  are  the  functional  communities  or  major  users  that  weapon  system  acquisition 
data  domains  support  within  the  DoD?  We  identified,  through  discussions  with  the 
information  managers  of  the  21  information  systems,  major  users  of  DoD  acquisition  data 
within  the  OSD. 

What  are  the  providers  of  weapon  system  acquisition  data  for  USD(AT&L) 
decision-making?  We  also  identified,  through  discussions  with  information  managers,  who 
is  providing  acquisition  data  to  OSD  information  systems  in  order  to  inform  USD(AT&L) 
decision-making  on  defense  acquisition. 
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What  are  some  recommendations  for  improving  the  acquisition  data 
environment?  In  this  task,  we  provide  recommendations  that  would  improve  the  quality  of 
acquisition  data,  ease  of  access,  efficiency  of  collection  and  use,  and  the  ability  to  link  data 
through  common  data  elements. 

Background  on  Acquisition  Data  in  the  Department  of  Defense 

Acquisition  data  and  information  take  on  a  wide  variety  of  forms  within  the 
Department  of  Defense  and  include  such  information  as  the  cost  of  weapon  systems  (both 
procurement  and  operations),  technical  performance,  contracts  and  contractor  performance, 
and  program  decision  memoranda.  These  data  can  be  characterized  as  both  “structured” 
and  “unstructured.”2  They  are  critical  to  the  management  and  oversight  of  the  $1.5  trillion 
portfolio  of  major  weapon  programs  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics. 

This  data  may  be  for  statutory,  regulation,  policy,  or  other  reasons.  DoDI  5000.02 
(2015,  Enclosure  1,  pp.  47-58)  provides  a  detailed  list  of  “statutory  and  regulatory 
requirements  at  each  of  the  milestones  and  other  decision  points  during  the  acquisition 
process.”  This  does  not  encompass  all  of  the  requirements,  but  is  a  centralized  source  for 
many  of  them.  Some  of  the  information  requirements  are  to  measure  cost,  schedule,  and 
performance  of  weapon  systems,  while  others  examine  testing,  cybersecurity,  requirements, 
budgeting,  alternatives,  and  technology  readiness. 

The  information  resides  throughout  the  DoD  at  all  levels,  from  program  offices  in  the 
Services  to  various  offices  within  OUSD(AT&L).  It  can  be  found  in  decentralized  locations 
(e.g.,  individual  computers)  and  centralized  locations  (e.g.,  information  systems).  The  DoD 
also  uses  data  that  reside  in  various  federal  information  systems.  There  is  a  plethora  of 
acquisition-related  data  sources  that  are  now  available.  The  data  elements  within  these 
information  systems  vary.  Some  data  elements3  are  unique  while  others  may  overlap, 
depending  on  different  definitions.  The  timeframes  for  the  various  data  elements  are  non¬ 
stationary,  meaning,  for  example,  that  one  information  system  has  data  from  1960  to 
current,  while  another  may  only  have  data  from  2010  to  current.  Acquisition  data  are  stored 
in  information  systems  with  differing  platforms  and  hardware:  architectures,  software,  and 
interfaces;  vendors;  and  databases.  There  is  varying  accessibility  and  security  requirements 
(depending  on  the  data  being  stored)  in  the  information  systems. 

Enterprise  Information  within  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  Acquisition  Resources  and  Analysis  Directorate 
categorizes  the  data  into  various  business  areas  including  Research  and  Development 


2  According  to  the  PC  Magazine  Online  Encyclopedia,  structured  data  are  “Data  that  can  be 
immediately  identified  within  an  electronic  structure  such  as  a  relational  database.”  Unstructured  data 
are  “Data  that  are  not  in  fixed  locations.  The  term  generally  refers  to  free-form  text  such  as  in  word 
processing  documents,  PDF  files,  e-mail  messages,  blogs,  Web  pages  and  social  sites”  (“Structured 
Data,”  n.d.-b). 

3  According  to  the  PC  Magazine  Online  Encyclopedia,  a  data  element  is  “The  fundamental  data 
structure  in  a  data  processing  system.  Any  unit  of  data  defined  for  processing  is  a  data  element;  for 
example,  ACCOUNT  NUMBER,  NAME,  ADDRESS  and  CITY.  A  data  element  is  defined  by  size  (in 
characters)  and  type  (alphanumeric,  numeric  only,  true/false,  date,  etc.).  A  specific  set  of  values  or 
range  of  values  may  also  be  part  of  the  definition”  (“Data  Element,”  n.d.-a). 
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(R&D),  Requirements,  Budget,  Contracting,  Contract  Performance,  Financial  Execution, 
Program  Cost/Schedule/Performance,  Human  Capital,  and  Acquisition  Oversight/Portfolio 
Management. 

Many  factors  affect  how  acquisition  data  is  collected  and  stored.  There  are  multiple, 
changing  conditions  that  affect  the  management  of  acquisition  data.  Information  owners  and 
managers  may  need  to  consider  whether  a  current  architecture  can  support  additional 
statutory  requirements,  administrative  changes,  or  security  policy  changes.  Technological 
advancements  may  also  be  implemented  to  improve 

•  Collection  efficiency 

•  Quality  of  the  data 

•  Aggregation  of  the  data 

•  Ease  of  access/use  of  the  information  system  and  its  data 

•  Analysis  of  data 

•  Archiving  data  for  future  analysis/education 

These  same  factors  can  also  affect  the  development  of  various  acquisition  systems. 
Acquisition  information  systems  were  created,  evolved,  or  repurposed  based  upon  data 
needs  and  legitimate  reasons  (e.g.,  statutory  needs).  They  have  been  developed  with 
varying  architectures  and  interfaces.  They  also  require  analysts  with  cross  system-analytic 
skills.  They  are  also  difficult  for  users  to  navigate  effectively,  and  can  takes  years  of 
consistent  access  and  use  to  fully  understand  and  master.  Most  systems  are  built  for 
reporting,  not  analysis.  Compliance  and  tracking  has  been  a  priority.  Acquisition  information 
systems  and  the  data  they  contain  may  be  designed  to  answer  today’s  current  questions, 
but  inflexible  to  answer  tomorrow’s  questions. 

This  analysis  found  that  there  are  also  barriers  to  the  use  of  each  system  and  cross 
use  between  the  information  systems.  Access  procedures  are  complicated  and  generally 
consist  of  many  steps  that  may  not  ultimately  guarantee  access.  There  are  varying  access 
procedures  and  permissions  between  and  sometimes  within  systems.  The  federal  systems 
have  much  data  available  to  the  public,  but  DoD  systems  are  mostly  restricted.  New  users 
can  have  great  difficulty  establishing  and  maintaining  access  (how  to,  where,  who,  what?). 
Full  access  to  acquisition  information  systems  enables  analysts  to  maximize  use  of  data. 

The  owners  and  managers  of  the  data  have  found  that  balancing  security  and  access  needs 
is  difficult. 

Background  and  Findings  on  Deep  Dives  of  Acquisition  Information  Systems 

As  part  of  this  effort  to  understand  acquisition  data  opportunities,4  we  conducted 
“deep  dives”  on  a  set  of  information  systems.  In  this  section,  we  summarize  the  information 
we  gathered  through  our  deep  dives.  We  reviewed  21  federal-wide,  OSD-level,  and  Service- 
level  information  systems  and  their  data  elements  in  order  to  identify  where  are  some  of  the 
acquisition  data  or  information  that  supports  current  requirements  in  DoDI  5000.02.  We 
reviewed  five  federal-level  information  systems,  12  OSD-level  information  systems,  and 


4  By  “data  opportunities,”  we  mean  identifying  data  that  can  potentially  be  used  for  analysis  of  various 
defense  acquisition  questions. 
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three  Service-level  systems  (one  Army,  one  Air  Force,  and  one  Navy).  Of  the  21  systems,  at 
least  one  study-team  member  had  previous  knowledge  of  1 1 .  For  five  systems,  a  study- 
team  member  had  limited  prior  or  current  knowledge;  and  for  the  final  five  systems,  no  one 
from  the  RAND  study  team  had  knowledge  from  use.  We  worked  with  our  sponsor,  ARA/EI, 
on  whether  to  pursue  access  to  the  information  systems  for  this  effort,  ultimately  deciding 
not  to  do  so. 

We  did  not  rely  exclusively  on  access  to  the  information  systems  in  order  to  conduct 
the  deep  dives.  We  also  collected  official  documentation  as  available,  and  requested 
additional  materials  from  those  managing  the  information  systems.  We  had  some  level  of 
open-source  materials  for  all  but  two  systems.  Finally,  we  relied  heavily  on  discussions  with 
the  information  managers,  particularly  on  the  information  systems  for  which  we  had  little  or 
no  knowledge  and  open-source  materials  were  not  available.  We  were  able  to  conduct 
discussions  for  all  but  one  information  system.  The  results  of  this  study  depend  on  the 
variety  of  information  we  were  able  to  collect. 

We  verified  the  deep-dive  information  with  information  managers  in  early  2016  in 
order  to  ensure  that  the  deep  dives  contain  the  latest  available  information.  Nevertheless, 
we  found  that  the  information  in  these  systems  is  constantly  changing  as  policy,  technology, 
and  other  things  change.  Consequently,  it  is  best  to  consult  the  information  systems  directly 
for  the  most  up-to-date  information. 

As  stated  previously,  we  gathered  additional  information  for  these  deep  dives 
through  discussions  with  information  managers.  The  information  that  we  gathered  from  the 
discussions  covered  the  following  main  topic  areas: 

•  Basic  details  on  the  acquisition  information  system 

•  Types  of  questions  answered  with  this  information  system 

•  Owner,  manager,  and  host  of  the  information  system  and  data  in  that 
information  system 

•  Statute  or  policies  that  led  to  the  creation  of  the  information  system  or  provide 
the  reason  the  data  in  the  system  is  collected 

•  Characterization  of  the  data  in  the  information  system 

•  Security  and  access  restrictions  governing  the  information  system 

•  Characterization  of  the  users 

•  Strengths  and  weaknesses  of  the  information  system  or  data  in  that 
information  system 

Basic  Details  on  the  Acquisition  Information  Systems 

For  each  of  the  21  information  systems,  we  gathered  basic  factual  information 
including  the  official  abbreviation,  date  that  the  system  entered  service,  the  access  point  for 
the  information  system,  whether  the  system  is  open  to  the  public  or  is  restricted,  the 
functional  business  area  the  system  supports,  and  the  purpose.  These  systems  cover  a 
wide  variety  of  functional  business  areas  including 

Research  and  development  (R&D) 

Requirements 
Budgeting 
Contracting 
Contract  Performance 
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•  Financial  Execution 

•  Program  Cost /Schedule/Performance 

•  Human  Capital 

•  Acquisition  Oversight/Portfolio  Management 

Some  systems  cover  multiple  business  areas. 

Types  of  Questions  Answered  by  These  Information  Systems 

Decision-makers  and  analysts  working  in  defense  acquisition  need  to  understand  the 
type  of  questions  that  can  be  answered  with  the  structured  and  unstructured  data  in  these 
information  systems.  They  also  need  to  know  what  questions  cannot  be  answered.  We 
asked  information  managers  to  identify  some  of  the  questions  that  can  be  answered  from 
the  data  in  these  information  systems. 

Owner,  Manager,  and  Host  of  the  Information  System 

Additional  factual  information  that  we  collected  on  these  information  systems 
included  the  owner,  manager,  and  host  of  these  systems.  The  owner  is  the  office 
responsible  for  oversight  of  the  information  system.  It  is  sometimes  different  from  the 
manager  of  the  system  who  may  be  responsible  for  day-to-day  operations  including 
approving  access  and  troubleshooting  technical  issues,  but  the  owner  and  manager  are 
typically  within  the  same,  larger  organization.  The  host  of  the  information  system  often 
appears  to  be  an  office  outside  of  the  owner  or  manager  and  is  typically  a  contractor  for  the 
federal  systems. 

Statute/Policies  Requiring  Each  Information  System 

Most  of  these  systems  originated  in  statute  requirements,  with  the  Federal 
Acquisition  Regulation  also  being  a  common  reason  for  creating  a  data  system.  Some 
systems  originated  in  policies  or  memoranda  from  senior  DoD  leadership. 

Characterization  of  the  Data  in  the  Information  System 

There  is  no  consensus  on  whether  the  data  in  these  systems  is  authoritative.  Some 
systems  contain  data  that  are  authoritative,  but  others  pull  data  from  elsewhere.  There  is 
also  significant  variation  in  the  dates  of  the  data  in  these  information  systems.  A  version  of 
one  information  system  goes  back  as  far  as  1951  for  the  DoD.  For  several  other  systems, 
there  may  be  some  historical  data  back  to  the  1960s.  Likewise,  there  is  some  variation  in 
whether  a  formal  data  dictionary  exists  and,  if  one  does,  whether  it  is  available  to  users.  In 
some  cases,  information  managers  use  the  data  dictionary  for  planning,  but  do  not  provide  it 
to  users.  In  some  systems,  data  elements  have  been  added  over  time  or  their  definitions 
have  changed. 

Characterization  of  the  Users 

The  number  of  users  for  these  information  systems  varied  from  less  than  100  to 
nearly  400,000  users.  Information  managers  may  count  their  users  as  “registered,”  “active,” 
“average  users  per  month,”  or  “number  of  users  in  a  particular  time  period.”  Composition  of 
users  also  varies  widely.  Some  of  the  information  managers  provided  high-level  statistics 
(e.g.,  public,  government,  DoD),  while  others  provide  specific  organization  names  for  users. 

Conclusions  and  Options 

Acquisition  data  and  information  take  on  a  wide  variety  of  forms  within  the 
Department  of  Defense  and  include  such  information  as  the  cost  of  weapon  systems  (both 
procurement  and  operations),  technical  performance,  contracts  and  contractor  performance, 
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and  program  decision  memoranda.  This  data  is  collected  for  a  variety  of  reasons  including 
statutory  requirements,  regulation,  policy,  and  other  reasons. 

The  information  resides  throughout  all  levels  of  the  DoD  and  can  be  found  in 
informal,  decentralized  locations  as  well  as  formal,  centralized  locations  (e.g.,  information 
systems).  The  DoD  also  uses  other  federal  data  residing  elsewhere. 

Data  elements  within  this  plethora  of  sources  may  vary.  Some  data  elements  are 
unique,  while  others  may  overlap  depending  on  different  definitions.  The  timeframe  and 
source  of  these  data  vary  as  well. 

There  are  multiple,  changing  conditions  that  affect  the  management  of  acquisition 
data.  Information  owners  and  managers  may  need  to  consider  whether  a  current 
architecture  can  support  additional  statutory  requirements,  administrative  changes,  or 
security  policy  changes.  Technological  advancements  may  also  be  implemented  to  improve 
collection  efficiency,  quality,  aggregation,  and  ease  of  access  or  use. 

These  same  conditions  can  also  affect  the  development  of  various  acquisition 
systems.  Acquisition  information  systems  were  created,  evolved,  or  repurposed  based  upon 
data  needs  and  legitimate  reasons  (e.g.,  statutory  needs).  Yet  they  are  often  difficult  for 
users  to  navigate  effectively  and  can  require  years  of  consistent  access  and  use  to  fully 
understand  and  master.  Most  systems  are  built  for  reporting,  not  analysis,  and  compliance 
and  tracking  has  been  a  priority.  Acquisition  information  systems  and  the  data  they  contain 
might  answer  current  questions  but  may  be  inflexible  for  future  ones. 

This  analysis  found  that  there  are  also  barriers  to  use  of  each  information  system 
and  cross  use  between  the  information  systems.  Access  procedures  are  complicated  and 
generally  have  many  steps  that  need  to  be  met  in  order  to  be  permitted  access  to  the 
information  system  and  its  comments.  There  are  also  varying  access  procedures/ 
permissions  between  and  sometimes  within  systems.  The  federal  systems  have  an 
abundance  of  data  available  to  the  public,  but  DoD  systems  are  mostly  restricted.  New 
users  can  have  great  difficulty  establishing  and  maintaining  access.  Although  full  access  to 
acquisition  information  systems  enables  analysts  to  maximize  use  of  data,  it  is  not  practical 
given  the  need  to  balance  security  and  access. 

Deep  Dive  Conclusions 

We  compiled  information  on  21  federal  and  DoD  information  systems  that  contain 
structured  and  unstructured  acquisition  data  and  information.  The  level  of  detail  we  were 
able  to  pull  together  on  each  information  system  and  its  contents  varied  considerably  based 
on 


•  RAND  team  user  experience  with  individual  systems 

•  Availability  and  access  to  official  policy  documentation  and  other  materials  on 
the  information  systems 

•  Interviewee  interpretation  of  discussion  questions 

There  was  a  wide  variety  of  interpretation  of  each  of  the  questions  in  the  interview 
protocol  and  how  these  questions  pertain  to  the  individual  information  systems  that  an 
information  manager  is  overseeing.  The  output  of  these  discussions  showed  that  even 
common  terms  like  “owner,”  “user,”  or  “data  element”  and  “data  dictionary”  are  subject  to 
interpretation,  which  suggests  that  a  common  taxonomy  would  be  difficult  to  implement,  but 
may  be  necessary.  Basic  details  were  fairly  easy  to  identify  and  verify.  We  also  pulled 
together  a  large  variety  of  potential  questions  that  can  be  answered  by  the  data  in  each 
information  system,  but  the  list  is  not  comprehensive  nor  an  assessment  of  how  well  the 
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questions  could  be  answered.  Nevertheless,  both  are  critical  information  for  decision¬ 
makers. 

Some  factual  information  can  be  difficult  to  assess,  given  subtle  distinctions  such  as 
those  between  owner  and  manager  in  some  cases.  In  other  cases,  it  was  easy  to  verify 
information  on  owners,  managers,  and  hosts,  because  all  three  functions  are  performed  by 
the  same  office.  Yet  some  owners,  managers,  and  hosts  changed  over  time,  so  it  was  not 
always  clear  who  held  which  role. 

The  list  of  policies  that  led  to  the  origins  of  these  systems  was  not  always  apparent 
as  some  of  the  systems  are  older,  some  systems  have  “morphed”  from  one  objective  to 
others,  and  there  has  been  a  turnover  in  personnel  who  manage  the  systems.  Some 
information  systems  provided  a  list  within  the  information  system  documenting  the  policies 
that  led  to  the  system  creation/the  data  in  the  system.  In  other  cases,  we  were  given  the 
information  during  our  discussions  with  information  managers. 

When  we  asked  about  security  and  access  and  the  user  base  with  information 
managers,  the  feedback  we  got  was  very  difficult  to  compare  across  systems.  Security  and 
access  were  intertwined  in  discussions  even  though  there  are  supposed  to  be  clear  origins 
in  statute  and  policy  that  require  both  security  and  access  restrictions.  Similarly,  the 
information  we  received  on  users  varied  by  number,  type,  and  characteristic. 

For  each  data  system  we  reviewed  we  also  sought  to  identify  strengths  and 
challenges  for  the  information  manager  and  users.  We  summarized  the  major  cross-cutting 
strengths  and  challenges  themes  associated  with  the  systems  reviewed.  The  following  are 
some  of  the  major  strengths: 

•  The  collection  and  standardization  of  selected  acquisition  related  information 
into  one  place  where  it  can  be  input,  accessed,  and  analyzed  by  those 
needing  to  use  it. 

•  Data  that  is  input  electronically  with  controls  (e.g.,  through  validation  checks 
and  business  rules)  to  assure  that  key  data  elements  are  entered,  edited,  and 
cross  checked  against  historical  and  other  data,  which  improves  data  quality. 

•  Systems  that  have  been  established  or  improved  to  answer  acquisition 
questions.  These  systems  are  attempting  to  pull  together  variables  in  one 
place  for  analysis,  so  as  to  improve  DoD  decision-making,  and  also  to  save 
funding  that  is  typically  spent  by  analysts  trying  to  cobble  together 
information. 

Information  managers  also  face  several  challenges  in  managing  acquisition  data, 
including  the  following: 

Data  quality  vary  depending  on  what  is  input  or  provided,  and  often  with  no 
means  to  verify  accuracy. 

The  need  to  have  the  originators  input  new  data  when  the  data  have 
changed. 

Assuring  access  to  those  who  need-to-know  while  protecting  sensitive  data. 
Access  procedures  vary  greatly  by  system,  burdening  those  needing  to 
access  multiple  systems. 

Inconsistency  in  terms.  The  same  term  can  have  different  meanings  in 
different  acquisition  systems  which  makes  analyses  across  systems 
particularly  challenging. 

Inconsistency  in  data  formats. 
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•  High  variance  in  hardware  and  software. 

•  Need  for  more  data  elements,  leveraging  of  authoritative  systems,  real  time 
editing  and  verification,  and  updating  to  new  platforms. 

•  Desired,  backlogged  improvements  and  sometimes  critical  updates  that  lack 
resources  for  implementation. 

Options  for  Improving  the  Acquisition  Data  Environment 

Our  analysis  yields  several  recommendations  for  improving  the  DoD  acquisition-data 
environment. 

Formalize  a  Data  Governance  and  Data  Management  Function 

To  answer  the  DoD’s  acquisition  questions,  the  USD(AT&L)  should  consider 
formalizing  a  data  management  and  governance  function  (e.g.,  data  steward)  to  oversee 
data  opportunities.  Any  decision  on  a  data  steward  would  need  to  consider  who  could  be  the 
authority  to  institutionalize/implement  these  changes  given  the  diversity  of  data  ownership  in 
the  DoD. 

Our  discussions  with  information  managers  and  our  literature  review  on  Master  Data 
Management  found  that  data  governance  plays  a  key  role  in  the  success  of  acquisition  data 
management.  In  particular,  data  governance  can  monitor  and  enforce  the  use  of  acquisition 
tools.  Data  governance  also  determines  the  process  and  structure  for  authority  control, 
planning,  monitoring,  and  enforcement  over  data  assets  (American  Institute  of  CPAs,  2013, 
p.  4).  While  data  quality/validation  focuses  on  managing  individual  pieces  of  data,  data 
governance  focuses  on  data  definitions,  policies,  and  processes,  including  those  for  data 
quality/validation.  Data  governance  has  two  primary  data-management  objectives:  planning 
and  supervision/control. 

A  data  steward  function  would  need  to  further  identify  where  and  what  data 
opportunities  exist  by  maintaining  a  master  list  of  data/information  and  authoritative  sources. 
As  can  be  seen  from  this  study,  authoritative  sources  are  not  always  integrated  into 
information  systems,  and  it  is  not  apparent  that  developers  have  a  good  understanding  of  all 
of  the  authoritative  sources.  There  appears  to  be  a  movement  in  that  direction,  but  the  DoD 
should  continue  to  re-syndicate  data  from  authoritative  sources. 

The  data  steward  and  information  managers  should  proactively  solicit  ways  to 
improve  value  of  the  data  from  all  categories  of  users  (inputters,  overseers,  and  analysts)  in 
order  to  improve  data  quality,  capability,  access,  usability,  and  functionality.  This  function 
could  also  improve  understanding  of  related  systems  and  identify  potential  opportunities  for 
consolidation. 

Improve  Data  Quality  and  Its  Analytic  Value 

The  DoD  should  require  that  all  new  systems  have  user  and  data  entry  guides  and 
data  dictionaries  that  describe  data  elements  and  their  sources  (e.g.,  another  system  or 
enterprise/personnel  entering).  This  informs  data  opportunities  and  may  eliminate 
duplication.  Information  managers  should  try  to  minimize  manual  entry  whenever  possible  or 
provide  validation  checks.  An  explicit  list  of  authoritative  sources  for  data  elements  should 
be  available  and  new  systems  should  be  required  to  use  them,  while  older  systems  migrate 
towards  them. 

Information  managers  frequently  mentioned  that  data  verification  and  validation  is  a 
top  priority  and  that  they  have  both  manual  and  automated  checks  built  into  the  systems. 
Information  managers  should  continue  and  expand  this  best  practice. 
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Information  managers  mentioned  one  of  their  challenges  is  to  be  able  to  continue  to 
update  their  systems  to  add  capability  and  comply  with  the  latest  security  requirements.  The 
DoD  should  require  system  owners  to  develop  and  update  plans  and  budgets  for  continuous 
improvement  of  data  quality  and  analytic  value,  and  document  unfunded  requirements 
linked  to  these  improvements. 

Make  Structured  Data  the  Top  Priority 

Current  practice  is  to  collect  DoD  Acquisition  data  in  structured  and  unstructured 
formats.  Both  types  of  formats  have  an  important  role  in  the  execution,  oversight,  and 
analysis  of  acquisition  programs.  However,  structured  data,  which  is  easier  to  use  for 
analysis,  should  be  the  top  priority.  The  DoD  should  minimize  the  use  of  unstructured  data, 
which  takes  more  resources  and  different  capabilities  to  make  useful  for  analysis.  More 
specifically,  structured  data 

•  allows  for  topic  metatags 

•  can  use  strategic  algorithms  to  check  quality 

•  maximizes  drop-down  menus;  minimizes  free  text 

Similarly,  a  large  amount  of  acquisition  information  is  produced  in  unstructured 
formats.  Since  not  all  data  can  be  converted  to  a  structured  format,  the  DoD  needs  to 
identify  ways  to  make  unstructured  data  more  useful.  Structured  data  is  easy  to  use  once 
meaning  and  access  has  been  determined. 

By  moving  toward  structured  data,  the  standardization  of  formats  for  acquisition  data 
would  promote  sharing  between  systems.  The  standardization  needs  to  take  into  account 
context  and  meaning  when  appropriate. 

Develop  and  Train  Organic  Capability  Among  the  DoD  Workforce  to  Use/Improve  Data 

RAND  has  spent  decades  using  acquisition  data  to  solve  difficult  questions  on  a 
variety  of  defense  acquisition  topics.  Answering  sophisticated  acquisition  questions  requires 
analysts  with  detailed  knowledge,  access,  and  experience  with  numerous  data  sets.  They 
also  need  knowledge  of  how  the  information  systems  and  their  data  have  changed  over  time 
to  do  trend  and  other  analyses.  When  utilizing  very  large  data  sets,  robust  processing  and 
storage  capacity  and  the  skills  of  research  programmers  are  critical. 

The  DoD  needs  to  ensure  that  its  workforce  is  educated  and  trained  to  fully 
understand,  analyze,  and  use  existing  acquisition  data  opportunities.  The  acquisition 
community  must  have  the  skills  and  aptitude  to  understand,  analyze,  and  use  this  data  to 
make  decisions.  Lastly,  but  importantly,  the  DoD  needs  to  continue  to  focus  on  developing 
internal,  organic  capability  to  use  and  improve  acquisition  data  to  better  understand  what 
data  is  being  collected,  what  data  should  be  collected,  and  how  that  information  can  inform 
DoD  decision-making. 
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Acquisition  Data  Have  Many  Purposes 


*  Transparency 

*  Cost  control 

*  Distribution  of  federal  spending 

*  Small  business  goals  attainment 

*  Identify  and  prevent  fraud,  waste,  and  abuse 

*  Analyses  for  improved  decision  making 

*  Compliance  and  tracking  items  in  various  processes 

*  Archive 
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Acquisition  Data  Opportunities 


•  AT&L/ARA/EI  has  identified  76  potential  acquisition  decision¬ 
making  data  opportunities 

*  These  data  opportunities  are  highly  dispersed  throughout  DoD 
and  the  Federal  Government 


Data  are  organized  by  both  information  system  owner  and  also  by 
the  following  functional  business  areas: 

-  Research  and  development  (R&D) 

-  Requirements 

-  Budgeting 


RAND 


Can  DoD  solve  its  acquisition  questions  with  these  sources? 


Contracting 

Contract  Performance 

Financial  Execution 

Program  Cost  /Schedule/Performance 

Human  Capital 

Acquisition  Oversight/Portfolio  Management 
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RAND  Was  Asked  to  Consider  Several  Key 

Questions 


*  What  data  are  available  to  help  assist  in  defense 
acquisition  decision-making?  (complete) 

*  Where  do  acquisition  data  reside?  (complete) 

*  Who  can  access  the  information?  (complete) 

*  Can  we  get  access  to  these  data  for  acquisition-related 
purposes?  (complete) 

*  What  would  we  utilize  the  data  for?  (future  work) 


RAND 


Approved  for  public  release;  distribution  is  unlimited. 


Doing  Data  Right  4 


Research  Approach 


Identify 

data 

domains 

Review 
information 
systems  and 
their  elements 


Identify 

data 

users 

Collect  user 
information 
from  systems 


Identify 

data 

providers 

Review  DoDI 
5000.02 
information 
requirements 


•  Heavily  leveraged  some  of  RAND’s  most  advanced  users  of  acquisition 
data  for  the  deep  dives 

•  Identified  policies  that  specified  the  origins  of  the  information  system  and 
data 

•  Reviewed  documents  that  identified  managers/owners/users/providers  of 
acquisition  data 

•  Conducted  discussions  to  supplement  RAND  knowledge  and  available 
information 

RAND  Doing  Data  Right  5 
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We  Analyzed  21  Acquisition  Information 

Systems 


•  We  analyzed  these  information  systems  in  order  to  identify  where 
acquisition  data  or  information  reside  that  support  current 
information  requirements  in  DoDI  5000.02 

•  We  reviewed  5  federal-level,  13  OSD-level,  and  3  service-level 
systems  (one  Army,  one  Air  Force,  and  one  Navy): 

-  Of  the  21  systems,  at  least  one  RAND  study-team  member  had 
working  knowledge  of  eleven; 

-  For  five  systems,  a  study-team  member  had  limited  prior  or 
current  working  knowledge;  and 

-  For  the  final  five  systems,  no  RAND  study  team  members  had 
working  knowledge 
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We  Pursued  Detailed  Information  on  Data  Opportunities 
Through  Discussions  With  Information  Managers 


Descriptive  details  on  systems 

Abbreviation/common  name 

Access  restrictions  by  data  type,  user,  and  security 
policy 

Full  data  information  system  name 

Data  source’s  openness  or  data  availability 

Date  entered  service 

“Tech  stack”  or  software  used  for  the  system 

Overview  url  for  entrance  (usually  work  best  with  which 
web  browser) 

Multiple  versions  because  of  access  or  attributes 

Functional  business  area(s)  data  supports 

Restrictions  on  downloading 

Purpose  of  the  system  and  its  data 

Process  for  requesting  access 

Owner,  manager,  and  host  of  the  system 

Data  elements  (e.g.,  unit  of  data  with  a  exact  meaning) 

Owner(s)  of  the  data  in  the  system 

Other  information  systems  that  this  system  feeds 

Organization  responsible  for  adding  or  populating  the 
system 

Types  of  questions  answered  using  the  data  in  this 
system 

Organization  who  developed  the  system 

Users:  number,  composition  (e.g.,  by  organization) 

Authoritative  sources  for  the  data  in  the  system 

Strengths  of  system  and  its  data 

Are  the  data  in  the  system  considered  authoritative 

Challenges  of  system  and  its  data 

Data  transmitted  to/from  the  system 

Recommendations  for  improving  system  and  its  data 

Policy  creating/managing  (e.g.,  statutory,  order,  policy 
directive) 

Discussions  showed  that  even  common  terms  like  “owner”  or  “user”  are  subject  to 
interpretation ,  which  suggests  that  a  common  taxonomy  would  be  difficult  to  implement. 
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Challenges  in  Conducting  This  Study 


•  Pursuing  access  to  information  systems:  RAND  worked  with 
ARA/EI  to  decide  on  systems  for  this  effort 

-  Given  some  challenges,  with  ARA/EI  agreement,  decided  not  to 
pursue  new  study  access  to  information  systems 

•  Access  varied  from  full  (11)  to  partial  (5)  to  no  access  (5) 

-  Relied  on  discussions  with  information  system  owners  and 
materials  available  on  the  systems 

-  Unable  to  verify  discussion  information  in  some  cases 

•  Substantial  time  required  (both  ARA/EI  and  RAND  time)  to  set  up 
discussions  with  information  managers 

-  Amount  of  information  obtained  from  discussions  varied 

•  Study  resources  were  primarily  used  to  identify  data  opportunities, 
rather  than  analyze/assess  data  quality 
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Plethora  of  Acquisition  Related  Data  Sources 


*  Formats:  structured  unstructured 

*  Elements:  unique  ->  overlapping 

-  Definitions  can  vary 

-  Some  driven  by  statutory  requirements 

*  Time  frames  are  non-stationary 

*  Systems  have  different: 

-  Platforms,  hardware 

-  Architectures,  software,  interfaces 

-  Vendors 

-  Databases 

*  Accessibility,  security  varies 
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Multiple  Factors 

Affect  Future  AQ  Data  Management... 


*  Current  architectures  may  not  support  additional 

-  Statutory  requirements 

-  Administrative  changes 

-  Security  policy  changes 

*  Technological  advancements  may  improve 

-  Collection  efficiency 

-  Quality 

-  Aggregation 

-  Ease  of  access/use 

*  Archiving  data  for  future  analysis/education 
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...and  Past  and  Future 
AQ  Systems  Development 


*  Statutory  compliance  reporting/management  tracking 
data  needs,  not  analysis,  largely  drove 

-  Creation 

-  Evolution 

-  Repurposing 

*  May  or  may  not  answer  today’s  questions,  but 
inflexible  to  answer  tomorrow’s 

*  Varying  architectures/interfaces  require  analysts  with 

-  Cross  system-analytic  skills 

-  Years  of  access/use  to  fully  understand/master 
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Also  Found  Barriers  to  Use ,  Cross  Use 


*  Access  procedures  are  complicated  and  generally  consist  of 
many  steps  that  may  or  may  not  guarantee  access  in  the  end 

*  Varying  access  procedures/permissions  between  and  sometimes 
within  systems 

-  Federal  systems  -  most  are  public 

-  DoD  systems  -  most  are  restricted  (some  more  than  others) 

-  New  users  can  have  great  difficulty  establishing  and 
maintaining  access  (how  to,  where,  who,  what?) 

*  Full  access  to  AQ  systems  enables  maximum  use  of  data  by 
analysts  with  a  need  to  know 

-  Balancing  security  and  access  needs  is  difficult 
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Lessons  Learned  From  RAND’s  Experience  Using 
Acquisition  Data  to  Answer  Complicated  Questions 


*  Answering  sophisticated  acquisition  questions  requires 
analysts  with  detailed  knowledge,  access,  and 
experience  with  numerous  data  sets 

*  When  utilizing  very  large  data  sets,  robust  processing 
and  storage  capacity  and  the  skills  of  research 
programmers  are  critical 

*  OSD  CAPE  and  ARA/EI  are  helping  to  support  improved 
access  and  use  of  available  acquisition  data 

-  There’s  a  focus  on  utilizing  authoritative  sources  and 
analytical  tools  within  CAPE  and  ARA/EI  systems 

-  ARA/EI  provides  DoD  acquisition  personnel  with  visibility  to 
data  opportunities  across  DoD 


RAND 


Approved  for  public  release;  distribution  is  unlimited. 


Doing  Data  Right  13 


Improving  Data  Quality  and  Analytic  Value 


*  Minimize  manual  entry  and  multiple  instances 

-  Designate  authoritative  data  element  sources 

-  Require  new  system  use  and  older  systems  migration 

*  Establish  ways  to  verify/validate  data  wherever  possible 

*  Require  development  of 

-  User/data  entry  guides 

-  Data  dictionaries  describing  data  elements/sources 

*  Require  system  owners  to 

-  Develop  and  update  plans  and  budgets  for  continuous 
improvement  of  data  quality  and  analytic  value 

-  Document  unfunded  requirements 

Who  has  the  authority  to  institutionalize/implement  these 
changes  given  the  diversity  of  data  ownership  in  DoD? 
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